Image processing system

ABSTRACT

Provided are an image processing method, an image processing system, an image processing device and a computer program for detecting a detection object such as nares of the driver with a high degree of accuracy in, for example, a system using an in-vehicle camera which is mounted on a vehicle and takes an image of the face of the driver. A detection object is diversified by a variety of detection methods such as a method for detecting a plurality of locations in the vertical direction as candidates during image pickup of an image, detecting a range to be a candidate of a detection object on the basis of the brightness of a pixel for each of rows of pixels lined up in the horizontal direction corresponding to each detected location and specifying a detection object from candidates of a detection object on the basis of the length of the detected range.

TECHNICAL FIELD

The present invention relates to: an image processing method fordetecting a specific detection object from a two-dimensional image inwhich pixels are lined up in a first direction and a second directionthat are different from each other; an image processing device to whichthe image processing method is applied; an image processing systemcomprising the image processing device; and a computer program forrealizing the image processing device, and in particular to an imageprocessing method, an image processing device, an image processingsystem and a computer program for enhancing the detection accuracy of adetection object.

BACKGROUND ART

Suggested as a device to support driving of a vehicle such as a car isan image processing device for taking an image of the face of the driverwith an in-vehicle camera, which is arranged at a location capable oftaking an image of the face of the driver, and performing imageprocessing to detect the location of the facial contour and the eyes ofthe driver from an obtained image (see Patent Document 1, for example).By using such a device, it is also possible to construct a system fordetecting the status of the driver and providing driving support such aswarning depending on the status of the driver such as inattentivedriving or drowsy driving. It should be noted that an automatic gainfunction of an in-vehicle camera is used for a certain level ofadjustment so that the brightness of an image to be obtained by imagepickup of the face of the driver becomes constant, as a state whereoutside light such as west sun streams down on the face of the driver inthe vehicle arises in a complicated manner and the illuminance of theface of the driver is not constant during driving.

[Patent Document 1] Japanese Patent Application Laid-Open No.2004-234367

DISCLOSURE OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

However, although an automatic gain function can deal with a case wherethe face of the driver is irradiated uniformly with outside light suchas sunlight or reflected light, the automatic gain function foradjusting the brightness of the entire image cannot deal with a localchange in the illuminance when radiation of outside light is notuniform. For example, when a partial change arises, i.e. when only aleft half of the face is irradiated with sunlight, there is a problemthat it is impossible to identify a dark part which is not irradiatedwith sunlight as the face and false detection arises, i.e., only abright part is detected as the facial contour. As described above, sincea light ray for irradiating the face of the driver in a running vehicleconstantly changes, there is a problem that a sufficient accuracy cannotbe obtained in detection of the face or each part of the face unless ajudgment is made comprehensively in diversified methods.

The present invention has been made in view of such circumstances, andit is the main object thereof to provide: an image processing method fordetecting a plurality of locations in the vertical direction ascandidates during image pickup in detection of a detection object suchas the nose of a person from an image obtained by a process such asimage pickup, detecting a range to be a candidate of a detection objecton the basis of the brightness of a pixel for each of rows of pixelslined up in the horizontal direction corresponding to each detectedlocation and specifying a detection object from candidates of adetection object on the basis of the length of a detected range, so asto diversify a detection method and enhance the detection accuracy; animage processing device to which the image processing method is applied;an image processing system comprising the image processing device; and acomputer program for realizing the image processing device.

Furthermore, another object of the present invention is to provide animage processing method, an image processing device, an image processingsystem and a computer program for detecting a detection object from theresult of addition based on the brightness of each other adjoining pixelto the brightness of one pixel and subtraction based on the brightnessof pixels located at a predetermined distance in the horizontaldirection and the vertical direction during image pickup from said onepixel, so as to diversify a detection method and enhance the detectionaccuracy.

Moreover, yet another object of the present invention is to provide animage processing method, an image processing device, an image processingsystem and a computer program for detecting a detection object as arange in the horizontal direction based on the result of integration inthe vertical direction of changes in the brightness of pixels lined upin the horizontal direction during image pickup and a range in thevertical direction based on the result of integration in the horizontaldirection of changes in the brightness of pixels lined up in thevertical direction, so as to diversify a detection method and enhancethe detection accuracy.

In addition, yet another object of the present invention is to providean image processing method, an image processing device, an imageprocessing system and a computer program for deciding the priority of adetection method on the basis of a mean value and a variance value ofthe brightness in order to select an effective detection methoddepending on the situation, so as to enhance the detection accuracy.

Means For Solving The Problems

An image processing method according to the first aspect is an imageprocessing method for detecting a specific detection object from atwo-dimensional image in which pixels are lined up in a first directionand a second direction that are different from each other, characterizedby comprising the steps of: integrating brightness of pixels lined up inthe first direction and deriving a change in an integrated value in thesecond direction; detecting a plurality of locations in the seconddirection as locations corresponding to candidates of a detection objecton the basis of the derived change in an integrated value; detecting arange in the first direction based on brightness of a pixel as acandidate of a detection object for each of rows of pixels lined up inthe first direction corresponding to each detected location; andspecifying a detection object from candidates of a detection object onthe basis of length of the detected range.

An image processing method according to the second aspect is an imageprocessing method for detecting a specific detection object from atwo-dimensional image in which pixels are lined up in a first directionand a second direction that are different from each other, characterizedby comprising the steps of: converting brightness of one pixel on thebasis of a result of addition based on brightness of each otheradjoining pixel and subtraction based on brightness of a pixel locatedat a predetermined distance in the first direction and brightness of apixel located at a predetermined distance in the second direction fromsaid one pixel; and detecting a detection object on the basis of aresult of conversion.

An image processing method according to the third aspect is an imageprocessing method for detecting a specific detection object from atwo-dimensional image in which pixels are lined up in a first directionand a second direction that are different from each other, characterizedby comprising the steps of: integrating numerical values, in the seconddirection, based on a change in brightness of pixels lined up in thefirst direction and deriving a change in an integrated value in thefirst direction; integrating numerical values, in the first direction,based on a change in brightness of pixels lined up in the seconddirection and deriving a change in an integrated value in the seconddirection; and detecting a detection object on the basis of a range inthe first direction based on the derived change in an integrated valuein the first direction and a range in the second direction based on thederived change in an integrated value in the second direction.

An image processing method according to the fourth aspect is an imageprocessing method for detecting a specific detection object from atwo-dimensional image including a plurality of pixels by a plurality ofdetection methods, characterized by comprising the steps of: computing amean value of brightness of a pixel; computing a variance value ofbrightness of a pixel; and deciding priority of a detection method onthe basis of the computed mean value and variance value.

An image processing device according to the fifth aspect is an imageprocessing device for detecting a specific detection object from atwo-dimensional image in which pixels are lined up in a first directionand a second direction that are different from each other, characterizedby comprising: deriving means for integrating brightness of pixels linedup in the first direction and deriving a change in an integrated valuein the second direction; candidate detecting means means for detecting aplurality of locations in the second direction as locationscorresponding to candidates of a detection object on the basis of thederived change in an integrated value; range detecting means fordetecting a range in the first direction based on brightness of a pixelas a candidate of a detection object for each of rows of pixels lined upin the first direction corresponding to each detected location; andspecifying means for specifying a detection object from candidates of adetection object on the basis of length of the detected range.

An image processing device according to the sixth aspect is the oneaccording to the fifth aspect, characterized in that the candidatedetecting means is constructed to detect a plurality of locations forwhich a change in an integrated value of brightness of pixels integratedin the first direction indicates a local minimum value.

An image processing device according to the seventh aspect is the oneaccording to the sixth aspect, characterized by comprising means forobtaining a quadratic differential value of the change in an integratedvalue derived by the deriving means, wherein the candidate detectingmeans is constructed to detect a predetermined number of locations in aplurality of locations indicating a local minimum value, on the basis ofa result of quadratic differential.

An image processing device according to the eighth aspect is the oneaccording to any one of the fifth aspect to the seventh aspect,characterized in that the range detecting means is constructed to detecta range on the basis of a change in brightness of pixels lined up in thefirst direction.

An image processing device according to the ninth aspect is the oneaccording to any one of the fifth aspect to the eighth aspect,characterized by further comprising means for detecting a range in thefirst direction of a detection area which includes a detection objectand has a range in the first direction larger than a detection object,wherein the specifying means is constructed to specify a detectionobject on the basis of a result of comparison between length of a rangein the first direction of a candidate of a detection object detected bythe range detecting means and length of a range in the first directionof a detection area including a detection object.

An image processing device according to the tenth aspect is the oneaccording to any one of the fifth aspect to the ninth aspect,characterized by further comprising: means for integrating brightness ofpixels lined up in the second direction according to the specifieddetection object and deriving a change in an integrated value in thefirst direction; local minimum value detecting means for detecting alocal minimum value from a change in an integrated value in the firstdirection; means for counting the number of detected local minimumvalues; and means for determining that the specified detection object isfalse when the counted number is smaller than a predetermined number.

An image processing device according to the eleventh aspect is the oneaccording to the tenth aspect, characterized by further comprising meansfor determining that the specified detection object is false when thenumber of continuous pixels in the second direction, which include apixel corresponding to the local minimum value detected by the localminimum value detecting means and have brightness equal to said pixel,is larger than a predetermined number.

An image processing device according to the twelfth aspect is an imageprocessing device for detecting a specific detection object from atwo-dimensional image in which pixels are lined up in a first directionand a second direction that are different from each other, characterizedby comprising: means for converting brightness of one pixel on the basisof a result of addition based on brightness of each other adjoiningpixel and subtraction based on brightness of a pixel located at apredetermined distance in the first direction and brightness of a pixellocated at a predetermined distance in the second direction from saidone pixel; and detecting means for detecting a detection object on thebasis of a result of conversion.

An image processing device according to the thirteenth aspect is the oneaccording to the twelfth aspect, characterized in that the detectingmeans is constructed to detect a pixel having a minimum converted valueas a detection object.

An image processing device according to the fourteenth aspect is animage processing device for detecting a specific detection object from atwo-dimensional image in which pixels are lined up in a first directionand a second direction that are different from each other, characterizedby comprising: first deriving means for integrating numerical values, inthe second direction, based on a change in brightness of pixels lined upin the first direction and deriving a change in an integrated value inthe first direction; second deriving means for integrating numericalvalues, in the first direction, based on a change in brightness ofpixels lined up in the second direction and deriving a change in anintegrated value in the second direction; and detecting means fordetecting a detection object on the basis of a range in the firstdirection based on the change in an integrated value in the firstdirection derived by the first deriving means and a range in the seconddirection based on the change in an integrated value in the seconddirection derived by the second deriving means.

An image processing device according to the fifteenth aspect is the oneaccording to the fourteenth aspect, characterized in that the firstderiving means is constructed to integrate indexes, in the seconddirection, based on a numerical value based on a brightness differencefrom an adjoining pixel in the first direction and a numerical valueindicating a low level of brightness of an adjoining pixel in the firstdirection and derive a change in an integrated value in the firstdirection, the second deriving means is constructed to integrateindexes, in the first direction, based on a numerical value based on abrightness difference from an adjoining pixel in the second directionand a numerical value indicating a low level of brightness of anadjoining pixel in the second direction and derive a change in anintegrated value in the second direction, and the detecting means isconstructed to detect a detection object on the basis of a range in thefirst direction from a location where the integrated value derived bythe first deriving means becomes maximum to a location where theintegrated value derived by the first deriving means becomes minimum anda range in the second direction from a location where the integratedvalue derived by the second deriving means becomes maximum to a locationwhere the integrated value derived by the second deriving means becomesminimum.

An image processing device according to the sixteenth aspect is an imageprocessing device for detecting a specific detection object from atwo-dimensional image including a plurality of pixels by a plurality ofdetection methods, characterized by comprising: means for computing amean value of brightness of a pixel; means for computing a variancevalue of brightness of a pixel; and means for deciding priority of adetection method on the basis of the computed mean value and variancevalue.

An image processing system according to the seventeenth aspect ischaracterized by comprising: an image processing device described in anyone of the fifth aspect to the sixteenth aspect; and an image pickupdevice for generating an image to be processed by the image processingdevice, wherein the detection object is an area including nares of aperson in an image taken by the image pickup device, the first directionis a horizontal direction, and the second direction is a verticaldirection.

A computer program according to the eighteenth aspect is a computerprogram for causing a computer to detect a specific detection objectfrom a two-dimensional image in which pixels are lined up in a firstdirection and a second direction that are different from each other,characterized by running: a procedure of causing a computer to integratebrightness of pixels lined up in the first direction and derive a changein an integrated value in the second direction; a procedure of causing acomputer to detect a plurality of locations in the second direction aslocations corresponding to candidates of a detection object on the basisof the derived change in an integrated value; a procedure of causing acomputer to detect a range in the first direction based on brightness ofa pixel as a candidate of a detection object for each of rows of pixelslined up in the first direction corresponding to each detected location;and a procedure of causing a computer to specify a detection object fromcandidates of a detection object on the basis of length of the detectedrange.

A computer program according to the nineteenth aspect is a computerprogram for causing a computer to detect a specific detection objectfrom a two-dimensional image in which pixels are lined up in a firstdirection and a second direction that are different from each other,characterized by running: a procedure of causing a computer to convertbrightness of one pixel on the basis of a result of addition based onbrightness of each other adjoining pixel and subtraction based onbrightness of a pixel located at a predetermined distance in the firstdirection and brightness of a pixel located at a predetermined distancein the second direction from said one pixel; and a procedure of causinga computer to detect a detection object on the basis of a result ofconversion.

A computer program according to the twentieth aspect is a computerprogram for causing a computer to detect a specific detection objectfrom a two-dimensional image in which pixels are lined up in a firstdirection and a second direction that are different from each other,characterized by running: a procedure of causing a computer to integratenumerical values, in the second direction, based on a change inbrightness of pixels lined up in the first direction and derive a changein an integrated value in the first direction; a procedure of causing acomputer to integrate numerical values, in the first direction, based ona change in brightness of pixels lined up in the second direction andderive a change in an integrated value in the second direction; and aprocedure of causing a computer to detect a detection object on thebasis of a range in the first direction based on the derived change inan integrated value in the first direction and a range in the seconddirection based on the derived change in an integrated value in thesecond direction.

A computer program according to the twenty first aspect is a computerprogram for causing a computer to detect a specific detection objectfrom a two-dimensional image including a plurality of pixels by aplurality of detection methods, characterized by running: a procedure ofcausing a computer to compute a mean value of brightness of a pixel; aprocedure of causing a computer to compute a variance value ofbrightness of a pixel; and a procedure of causing a computer to decidepriority of a detection method on the basis of the computed mean valueand variance value.

In the first aspect, the fifth aspect, the sixth aspect, the seventhaspect and the eighteenth aspect, when the detection object is, forexample, an area including nares in the face of a person to right andleft nostrils in a taken image and the first direction and the seconddirection are respectively the horizontal direction and the verticaldirection, it is possible to detect a detection object with a highdegree of accuracy by focusing on the brightness distribution in thevertical direction, detecting a plurality of candidates including eveneyebrows, eyes and a mouth and specifying a detection object fromcandidates on the basis of the width in the horizontal direction.Especially, when being used in combination with another method fordetecting a detection object on the basis of the brightness distributionin the horizontal direction, it is possible to further enhance thedetection accuracy since it is possible to diversify a detection methodand judge the location of the detection object comprehensively.

In the eighth aspect, it is possible to clarify a difference in therange in the horizontal direction between an area including nares andother part and enhance the detection accuracy, by detecting a range onthe basis of the brightness of pixels lined up in the horizontaldirection from a plurality of candidates including parts such aseyebrows, eyes and a mouth.

In the ninth aspect, it is possible to enhance the detection accuracy bydetecting the width of the face, which is a detection area includingnares that is a detection object, and specifying a candidate, for whichcomparison to the width of the face, e.g. a range in the horizontaldirection to the width of the face, is 22% -43%, as a detection object,since it is possible to clearly distinguish from other parts such aseyebrows, eyes and a mouth.

In the tenth aspect, it is possible to reduce the possibility of falsedetection of judging a part other than nares as nares, by determiningthat the possibility of nares is low and a detection object is notdetected when the number of local minimum values in the horizontaldirection, i.e. the number of parts having high possibility of nares, issmaller than a predetermined number such as two which is set for rightand left nares.

In the eleventh aspect, it is possible to reduce the possibility offalse detection, by determining that it is highly possible that aneyeglass frame is detected and a detection object is not detected whenthe continuity in the vertical direction is larger than a thresholdindicated by a predetermined number.

In the second aspect, the twelfth aspect, the thirteenth aspect and thenineteenth aspect, when the detection object is, for example, nares inthe face of a person in a taken image and the first direction and thesecond direction are respectively the horizontal direction and thevertical direction, it is possible to detect a detection object with ahigh degree of accuracy by performing a conversion process foremphasizing a part, for which adjoining pixels have low brightness andthe surrounding thereof has high brightness, i.e., a small area havinglow brightness. Especially, when being used in combination with anothermethod for detecting a detection object, it is possible to furtherenhance the detection accuracy since it is possible to diversify adetection method and judge the location of the detection objectcomprehensively.

In the third aspect, the fourteenth aspect, the fifteenth aspect and thetwentieth aspect, when the detection object is, for example, an areasurrounding nares in the face of a person in a taken image and the firstdirection and the second direction are respectively the horizontaldirection and the vertical direction, it is possible to detect adetection object with a high degree of accuracy by deriving a numericalvalue, which becomes large at a position where the descent of thebrightness is large and becomes small at a position where the ascent ofthe brightness is large, for each of the vertical direction and thehorizontal direction and detecting a detection object which is aquadrangular area having lower brightness than the surrounding on thebasis of the derived numerical value. Especially, since detected is notnares themselves but a downside area surrounding nares, it is possibleto detect a detection object even when the face is inclined at an anglewhich makes detection of nares difficult. Especially, when being used incombination with another method, it is possible to further enhance thedetection accuracy since it is possible to diversify a detection methodand judge the location of the detection object comprehensively.Moreover, when a part such as the location of eyes or the location of anose is detected by another method, it is possible to detect a detectionobject with a higher degree of accuracy by performing detectionaccording to the present invention after narrowing down an area on thebasis of the positional relationship to the detected part.

In the fourth aspect, the sixteenth aspect and the twenty first aspect,when a detection object such as, for example, nares in the face of aperson is detected in a taken image, it is possible to further enhancethe detection accuracy by determining whether a local change in theilluminance is generated in the face of a person or not on the basis ofa mean value and a variance value of the brightness and deciding thepriority of a detection method depending on the determined situation,since a highly reliable detection method and detection prioritydepending on the situation can be selected from a variety of detectionmethods.

The seventeenth aspect, which can detect an area including nares of aperson with a high degree of accuracy, can be applied to, for example, asystem for detecting the face of the driver as a detection object froman image obtained by image pickup of the face of the driver with animage pickup device such as an in-vehicle camera mounted on a vehicleand developed to a system for detecting the status of the driver andproviding driving support such as warning to inattentive driving.

EFFECTS OF THE INVENTION

An image processing method, an image processing device, an imageprocessing system and a computer program according to the presentinvention are applied, for example, to an embodiment for a detectionobject of an area including nares in the face of a person to right andleft nostrils from an image obtained by image pickup of the face of thedriver with an image pickup device such as an in-vehicle camera mountedon a vehicle. In addition, an image processing device and the likeaccording to the present invention integrate the brightness of pixelslined up in the horizontal direction, which is the first direction, andderive a change in an integrated value in the vertical direction, whichis the second direction; detect a plurality of locations to be a localminimum value from a change in the derived integrated value ascandidates including a detection object; obtain a quadratic differentialvalue of a change in an integrated value and further narrow downcandidates therefrom to a predetermined number; detect a range in thefirst direction to be a candidate of a detection object on the basis ofthe brightness of a pixel for each of rows of pixels lined up in thefirst direction corresponding to each narrowed-down location; andspecify a detection object from candidates of a detection object on thebasis of the length of the detected range.

With this structure, the present invention guarantees beneficial effectssuch as detection of a detection object with a high degree of accuracyby focusing on the brightness distribution in the vertical direction,detecting a plurality of candidates including even eyebrows, eyes and amouth having low brightness and specifying a detection object fromcandidates on the basis of the width in the horizontal direction.Especially, when being used in combination with another method fordetecting a detection object on the basis of the brightness distributionin the horizontal direction, the present invention guarantees beneficialeffects such as further enhancement of the detection accuracy since itis possible to diversify a detection method and judge the location ofthe detection object comprehensively.

In addition, when being applied to a system for precisely detecting thestatus of the driver and providing driving support such as warning toinattentive driving by enhancing the accuracy of detection of adetection object, the present invention guarantees beneficial effectssuch as construction of a reliable driving support system which makesless false detection even in driving in an environment wherein thestatus of outside light constantly changes.

Moreover, an image processing device and the like according to thepresent invention guarantee beneficial effects such as detection toclarify a difference of a range in the horizontal direction between anarea including nares and other parts such as eyebrows, eyes and a mouthand enhancement of the detection accuracy as a result, by detecting arange on the basis of a change in the brightness of pixels lined up inthe first direction in detection of a range to be a candidate of adetection object from pixel rows lined up in the horizontal directionwhich is the first direction, and in particular, by detecting a range onthe basis of the result of detection of both ends of pixel rows havinglow brightness lined up in the first direction by a filtering processfor emphasizing a position where the brightness in the horizontaldirection rapidly changes.

Furthermore, an image processing device and the like according to thepresent invention guarantee beneficial effects such as enhancement ofthe detection accuracy by detecting the width of the face and specifyinga candidate, for which comparison to the width of the face, e.g. a rangein the horizontal direction to the width of the face, is 22% -43%, as adetection object, since it is possible to clearly distinguish from otherparts such as eyebrows, eyes and a mouth.

In addition, an image processing device and the like according to thepresent invention guarantee beneficial effects such as reduction of thepossibility of false detection of judging a part other than nares asnares, by determining that the possibility of nares is low and adetection object is not detected when the number of parts having lowerbrightness than the surrounding, for which an integrated value of thebrightness in the vertical direction according to a detection objectbecomes a local minimum value, i.e. the number of parts having highpossibility of nares, in a row of pixels lined up in the horizontaldirection included in a specified detection object is smaller than apredetermined number, i.e. two, which indicates the number of right andleft nares, even when a detection object can be specified.

Furthermore, an image processing device and the like according to thepresent invention guarantee beneficial effects such as reduction of thepossibility of false detection by judging that it is highly possiblethat an eyeglass frame is detected and a detection object is notdetected when the number of pixels lined up in the vertical direction islarger than a predetermined number.

An image processing method, an image processing device, an imageprocessing system and a computer program according to the presentinvention are applied, for example, to an embodiment for a detectionobject of nares in the face of a person from an image obtained by imagepickup of the face of the driver with an image pickup device such as anin-vehicle camera mounted on a vehicle. In addition, an image processingdevice and the like according to the present invention convert thebrightness of one pixel on the basis of the result of addition based onthe brightness of each other adjoining pixel and subtraction based onthe brightness of a pixel located at a predetermined distance in thehorizontal direction, which is the first direction, and a pixel locatedat a predetermined distance in the vertical direction, which is thesecond direction, from the location of said one pixel; and detect apixel having a minimum converted value as a detection object.

With this structure, the present invention guarantees beneficial effectssuch as detection of nares, which are a small area having lowbrightness, with a high degree of accuracy by performing a conversionprocess for emphasizing a part, for which adjoining pixels have lowbrightness and the surrounding thereof has high brightness, i.e., asmall area having low brightness. Especially, when the present inventionwhich can detect nares with a high degree of accuracy while the face isturned up is used in combination with another method for detecting adetection object, the present invention guarantees beneficial effectssuch as further enhancement of the detection accuracy since it ispossible to diversify a detection method and judge the location of thedetection object comprehensively.

In addition, when being applied to a system for precisely detecting thestatus of the driver and providing driving support such as warning toinattentive driving by enhancing the accuracy of detection of adetection object, the present invention guarantees beneficial effectssuch as construction of a reliable driving support system which makesless false detection even in driving in an environment wherein thestatus of outside light constantly changes.

An image processing method, an image processing device, an imageprocessing system and a computer program according to the presentinvention are applied, for example, to an embodiment for a detectionobject of a downside area surrounding nares in the face of a person froman image obtained by image pickup of the face of the driver with animage pickup device such as an in-vehicle camera mounted on a vehicle.In addition, an image processing device and the like according to thepresent invention integrate indexes, which are obtained by multiplying anumerical value based on a brightness difference from an adjoining pixelin the horizontal direction that is the first direction by a numericalvalue indicative of a low level of brightness of an adjoining pixel inthe first direction, in the vertical direction which is the seconddirection and derive a change in an integrated value in the firstdirection; integrate indexes, which are obtained by multiplying anumerical value based on a brightness difference from an adjoining pixelin the second direction by a numerical value indicative of a low levelof brightness of an adjoining pixel in the second direction, in thefirst direction and derive a change in an integrated value in the seconddirection; and detect a detection object on the basis of a range in thefirst direction and a range in the second direction from a locationwhere the derived integrated values respectively become maximum to alocation where the derived integrated values respectively becomeminimum.

With this structure, the present invention guarantees beneficial effectssuch as detection of a detection object with a high degree of accuracysince a numerical value which becomes large at a position where thedescent of the brightness is large and becomes small at a position wherethe ascent of the brightness is large is derived for each of thehorizontal direction and the vertical direction and a detection objectwhich is a quadrangular area having lower brightness than thesurrounding is detected on the basis of the derived numerical value.Especially, since detected is not nares themselves but a downside areasurrounding nares as a quadrangular area having lower brightness thanthe surrounding, the present invention guarantees beneficial effectssuch as detection of a detection object even when the face is inclinedat an angle which makes detection of nares difficult. Furthermore, whena part such as the location of eyes and the location of a nose isdetected by another method, the present invention guarantees beneficialeffects such as detection of a detection object with a higher degree ofaccuracy by performing detection according to the present inventionafter narrowing down an area on the basis of the positional relationshipto the detected part. Accordingly, when being used in combination withanother method for detecting a detection object, the present inventionguarantees beneficial effects such as further enhancement of thedetection accuracy since it is possible to diversify a detection methodand judge the location of the detection object comprehensively.

In addition, when being applied to a system for precisely detecting thestatus of the driver and providing driving support such as warning toinattentive driving by enhancing the accuracy of detection of adetection object, the present invention guarantees beneficial effectssuch as construction of a reliable driving support system which makesless false detection even in driving in an environment wherein thestatus of outside light constantly changes.

An image processing method, an image processing device, an imageprocessing system and a computer program according to the presentinvention are applied, for example, to an embodiment for a detectionobject of an area including nares in the face of a person from an imageobtained by image pickup of the face of the driver with an image pickupdevice such as an in-vehicle camera mounted on a vehicle. In addition,an image processing device and the like according to the presentinvention compute a mean value of the brightness of a pixel, compute avariance value of the brightness of a pixel and decide the priority of adetection method from a plurality of detection methods on the basis ofthe computed mean value and variance value.

With this structure, the present invention guarantees beneficial effectssuch as further enhancement of the detection accuracy since it ispossible to select a highly reliable detection method and detectionpriority depending on the situation from a variety of detection methodsby determining whether a local change in the illuminance is generated inthe face of a person or not on the basis of a mean value and a variancevalue of the brightness and deciding the priority of a detection methoddepending on the determined situation.

BRIEF DESCRIPTION OF THE DRAWINGS

Drawing 1 is a block diagram showing a structure example of an imageprocessing system according to Embodiment 1 of the present invention;

Drawing 2 is a flow chart showing an example of a process of an imageprocessing device to be used in an image processing system according toEmbodiment 1 of the present invention;

Drawing 3 is a flow chart showing an example of a process of an imageprocessing device to be used in an image processing system according toEmbodiment 1 of the present invention;

Drawing 4 is an explanatory view conceptually showing an example of aprocess from decision of a range to detection of a candidate of adetection object of a process of an image of an image processing systemaccording to Embodiment 1 of the present invention;

Drawing 5 is an explanatory view schematically showing an example of arange for which an end detecting process of an image processing systemaccording to Embodiment 1 of the present invention is performed;

Drawing 6 is an explanatory view showing an example of coefficients tobe used in an end detecting process of an image processing systemaccording to Embodiment 1 of the present invention;

Drawing 7 is an explanatory view schematically showing an example of arange for which an end detecting process of an image processing systemaccording to Embodiment 1 of the present invention is performed;

Drawing 8 is an explanatory view schematically showing an example of arange for which an end detecting process of an image processing systemaccording to Embodiment 1 of the present invention is performed;

Drawing 9 is an explanatory view showing candidates of a detectionobject of an image processing system according to Embodiment 1 of thepresent invention;

Drawing 10 is an explanatory view conceptually showing a naris areascore of an image processing system according to Embodiment 1 of thepresent invention;

Drawing 11 is a block diagram showing a structure example of an imageprocessing system according to Embodiment 2 of the present invention;

Drawing 12 is a flow chart showing an example of a process of an imageprocessing device to be used in an image processing system according toEmbodiment 2 of the present invention;

Drawing 13 is an explanatory view conceptually showing an example ofsetting of a detection range of an image processing system according toEmbodiment 2 of the present invention;

Drawing 14 is an explanatory view conceptually showing an example ofsetting of a search range of an image processing system according toEmbodiment 2 of the present invention;

Drawing 15 is an explanatory view showing an example of coefficients tobe used in a black area computation filtering process of an imageprocessing system according to Embodiment 2 of the present invention;

Drawing 16 is an explanatory view conceptually showing an example ofdetection using a black area computation filtering process of an imageprocessing system according to Embodiment 2 of the present invention;

Drawing 17 is a block diagram showing a structure example of an imageprocessing system according to Embodiment 3 of the present invention;

Drawing 18 is a flow chart showing an example of a process of an imageprocessing device to be used in an image processing system according toEmbodiment 3 of the present invention;

Drawing 19 is an explanatory view conceptually showing an example ofsetting of a search range of an image processing system according toEmbodiment 3 of the present invention;

Drawing 20 is an explanatory view showing an example of coefficients tobe used in a horizontal edge filtering process of an image processingsystem according to Embodiment 3 of the present invention;

Drawing 21 is an explanatory view showing an example of coefficients tobe used in a vertical edge filtering process of an image processingsystem according to Embodiment 3 of the present invention;

Drawing 22 is an explanatory view showing a result of detection of animage processing system according to Embodiment 3 of the presentinvention;

Drawing 23 is a block diagram showing a structure example of an imageprocessing system according to Embodiment 4 of the present invention;and

Drawing 24 is a flow chart showing an example of a process of an imageprocessing device 2 to be used in an image processing system accordingto Embodiment 4 of the present invention.

DESCRIPTION OF THE NUMERALS

1 Image Pickup Device

2 Image Processing Device

31, 32, 33, 34 Computer Program

41, 42, 43, 44 Record Medium

BEST MODE FOR IMPLEMENTING THE INVENTION

The following description will explain the present invention in detailwith reference to the drawings illustrating some embodiments thereof.

Embodiment 1

Drawing 1 is a block diagram showing a structure example of an imageprocessing system according to Embodiment 1 of the present invention.Denoted at 1 in Drawing 1 is an image pickup device such as anin-vehicle camera mounted on an vehicle, which is connected with animage processing device 2 for performing image processing via acommunication network such as an in-vehicle LAN (Local Area Network)having a wired or wireless structure, or a communication line such as adedicated cable. The image pickup device 1 is disposed in front of thedriver, e.g. at the handle or the dashboard in a vehicle, and adjustedto be able to take an image so that the width and the height of the faceof the driver become the horizontal direction and the vertical directionof an image.

The image pickup device 1 comprises: an MPU (Micro Processor Unit) 11for controlling the entire device; a ROM (Read Only Memory) 12 forrecording data and various kinds of computer programs to be executedbased on control of the MPU 11; a RAM (Random Access Memory) 13 forrecording various kinds of data to be generated temporally duringexecution of a computer program recorded in the ROM 12; an image pickupunit 14 constituted of an image pickup element such as a CCD (ChargeCoupled Device); an A/D converter 15 for converting analog image dataobtained by image pickup of the image pickup unit 14 into digital data;a frame memory 16 for temporally recording digital image data convertedby the A/D converter 15; and a communication interface 17 to be used forcommunication with the image processing device 2.

In the image pickup device 1, the image pickup unit 14 performs an imagepickup process continuously or intermittently so as to generate 30 imagedata (image frames) per second, for example, on the basis of the imagepickup process and output the same to the A/D converter 15, and the A/Dconverter 15 converts each pixel constituting an image into digitalimage data which is shown by a gradation such as 256 gradation (1 Byte)and causes the frame memory 16 to record the digital image data. Theimage data recorded by the frame memory 16 is outputted from thecommunication interface 17 to the image processing device 2 atpredetermined timing. Each pixel constituting an image is arranged in atwo-dimensional manner and the image data includes data indicative ofthe location of each pixel indicated by a plane rectangular coordinatesystem, or what is called an XY coordinate system, and the brightness ofeach pixel indicated as a gray value. It should be noted that acoordinate of each pixel may be respectively indicated by the order tobe arranged in data, instead of an XY coordinate system. In addition,the horizontal direction of an image corresponds to the X-axis directionof image data and the vertical direction of an image corresponds to theY-axis direction of image data.

The image processing device 2 comprises: a CPU (Central Processing Unit)21 for controlling the entire device; an auxiliary recording unit 22such as a CD-ROM drive for reading information from a record medium 41such as a CD-ROM for recording various kinds of information such as dataand a computer program 31 according to Embodiment 1 of the presentinvention; a hard disk (which will be hereinafter referred to as an HD)23 for recording various kinds of information read by the auxiliaryrecording unit 22; a RAM 24 for recording various kinds of data to begenerated temporarily during execution of the computer program 31recorded in the HD 23; a frame memory 25 constituted of a volatilememory; and a communication interface 26 to be used for communicationwith the image pickup device 1.

In addition, by reading various kinds of information such as data andthe computer program 31 of the present invention from the HD 23, causingthe RAM 24 to record the same and running various kinds of proceduresincluded in the computer program 31 with the CPU 21, an in-vehiclecomputer operates as the image processing device 2 of the presentinvention. Data to be recorded in the HD 23 includes data such as dataaccording to execution of the computer program 31, a variety of datasuch as mathematical expressions, which will be explained later, filtersand various kinds of constants, for example, and also data indicative ofa detected detection object or a candidate of a detection object.

In the image processing device 2, image data outputted from the imagepickup device 1 is accepted at the communication interface 26, theaccepted image data is recorded in the frame memory 25 and the imagedata recorded in the frame memory 25 is read out for performing avariety of image processing. A variety of image processing to beexecuted for the accepted image data is a variety of processes necessaryfor detection of an area such as the facial contour, eyes and a nose ofthe driver from image data. A concrete example of a process is a contourwidth detecting process of integrating the brightness on a vertical lineof an image and comparing the integrated value with a predeterminedthreshold so as to detect the range in the horizontal direction of thefacial contour constituted of pixels having higher brightness thanbackground. Moreover, another example of a process is a contour widthdetecting process of further differentiating a change in the horizontaldirection in the integrated value in the above process to identify thelocation having a large change and detecting a boundary betweenbackground and the facial contour where the brightness greatly changes.The detailed content of the processes is described in documents such asJapanese Patent Application Laid-Open Nos. 2004-234494 and 2004-234367,for example, which have been applied by the present applicant. It shouldbe noted that the above image processing is not always limited to theprocesses described in Japanese Patent Application Laid-Open Nos.2004-234494 and 2004-234367, and can be selected properly according tothe terms such as the use thereof, the hardware structure or cooperationwith other application programs.

The following description will explain a process of various kinds ofdevices to be used in an image processing system according to Embodiment1 of the present invention. Embodiment 1 of the present invention is fora detection object of an area including nares in the face of the driverto right and left nostrils from an image obtained by image pickup of theface of the driver with the image pickup device 1 such as an in-vehiclecamera mounted on a vehicle, for example, as explained using Drawing 1.Drawing 2 and Drawing 3 is a flow chart showing an example of a processof the image processing device 2 to be used in an image processingsystem according to Embodiment 1 of the present invention. The imageprocessing device 2 extracts image data, which is obtained by imagepickup of the image pickup device 1 and accepted via the communicationinterface 26, from the frame memory 25 under control of the CPU 21 forexecuting the computer program 31 recorded in the RAM 24 (S101), detectsthe width of the face of the driver, i.e. a range in the horizontaldirection (first direction) of the contour which is the boundary of anarea indicative of the face, from the extracted image data by, forexample, the contour width detecting process described above (S102) andsets a range of image processing to be performed later on the basis ofthe detected result (S103). The range (width of the contour) of an areadetected in the step S102 and the range of image processing are recordedin the HD 23 or the RAM 24.

The image processing device 2 then integrates the brightness of pixelslined up in the horizontal direction (first direction) for an image ofthe range, which is extracted in the step S101 and set in the step S103,under control of the CPU 21 (S104), derives a change in an integratedvalue in the vertical direction (second direction) from the integratedresult (S105) and detects a plurality of locations indicative of a localminimum value as locations in the vertical direction corresponding tocandidates of a detection object, from the derived change in anintegrated value in the vertical direction (S106). Detected in the stepS106 is not only an area including nares, which are the originaldetection object, to right and left nostrils but also a plurality ofcandidates including even eyebrows, eyes and a mouth having lowbrightness.

The image processing device 2 obtains a quadratic differential value ofa change in the integrated value, which is derived in the step S105,under control of the CPU 21 (S107) and detects a predetermined numberof, such as ten, locations as locations of candidates including adetection object in the vertical direction in order of increasingquadratic differential value in a plurality of locations indicative of alocal minimum value detected in the step S106 (S108). The process in thestep S107 for obtaining a quadratic differential value of a change inthe integrated value is performed using the following Expression 1, forexample. The step S107 is a process for narrowing down candidatesdetected in the step S106, and up to ten candidates are detected in theprocess in the steps S104-S108. Here, it will be obvious that the numberof candidates to be detected in the step S108 becomes smaller than tenwhen the number of candidates detected in the step S106 is smaller thanten. It should be noted that the predetermined number is a numericalvalue which can be changed according to need, which is preliminarilyrecorded in the HD 23 or the RAM 24. Moreover, data indicative ofcandidates detected in the step S108 is recorded in the HD 23 or the RAM24.Quadratic Differential Value=P(y)·2−P(y−8)−P(y+8)   Expression 1

Here, y: coordinate in the vertical direction (y coordinate) P(y):integrated value of location y

-   -   The image processing device 2 then reads each location in the        vertical direction, which is a candidate of a detection object        detected in the step S108, from the HD 23 or the RAM 24 under        control of the CPU 21, detects ends in the horizontal direction,        i.e. the right and left ends, of a candidate of a detection        object for each of rows of pixels lined up in the horizontal        direction corresponding to each read location on the basis of a        change in the brightness of pixels (S109) and detects a range in        the horizontal direction to be a candidate of a detection object        on the basis of the detected right and left ends (S110). It        should be noted that a candidate the right and left ends of        which are not detected in the step S109, i.e. a candidate only        one end side of which is detected or a candidate no end of which        is detected, is excluded from candidates of a detection object        and the content recorded in the HD 23 or the RAM 24 is updated.        The process in the step S109 is performed by detecting a point,        where the continuous state of pixels having lower brightness        than the surrounding ends, as an end as will be explained later.        Moreover, the process in the steps S109-S110 is performed for        all candidates of a detection object and further narrowing down        of candidates of a detection object and detection of a range are        performed.

The image processing device 2 then compares the length of a range in thehorizontal direction of a candidate of a detection object, which isdetected in the step S110 and recorded, with the length of the range(width of contour) in the horizontal direction of an area of the face ofthe driver, which is detected in the step S102 and recorded, undercontrol of the CPU 21, specifies a candidate of a detection object, forwhich the ratio of the length of the range in the horizontal directionto the length of the range in the horizontal width in the verticaldirection of a predetermined number of, such as three or five, pixels,under control of the CPU 21 (S112), integrates the brightness of pixelslined up in the vertical direction in the set test area, derives achange in an integrated value in the horizontal direction (S113),detects local minimum values from the derived change in an integratedvalue in the horizontal direction (S114), counts the number of thedetected local minimum values (S115) and determines whether the countednumber of local minimum values is smaller than a predetermined number,two here, 10 or not (S116). When the number of local minimum valuescounted in the step S115 is smaller than the predetermined number (S116:YES), the image processing device 2 determines that the detection objectspecified in the step S111 is false and detection of a detection objectis impossible under control of the CPU 21 (S117), updates the recordcontent of the HD 23 or the RAM 24 on the basis of the determined resultand terminates the process. When the specified detection object is anarea including nares to right and left nostrils, it is possible to counttwo or more local minimum values since locations corresponding to nareshave a local minimum value in a change in an integrated value derived inthe step S113. Accordingly, when the number of local minimum values iszero or one, it is determined that the specified specified detectionobject is false. It should be noted that, even when there are three ormore local minimum values, it is not determined that the detectionobject is false only by the fact since a location such as the vicinityof the contour of nostrils becomes shady and may indicate a localminimum value.

When the number of the counted local minimum values is two, which is thepredetermined number, or larger in the step S116 (S116: NO), the imageprocessing device 2 counts the number of pixels continuous in thevertical direction, which include a pixel corresponding to the localminimum values and have brightness equal to said pixel, under control ofthe CPU 21 (S118) and determines whether the counted number ofcontinuous pixels is larger than or equal to a predetermined number,which is preliminarily set, or not (S119). In the step S118, thecontinuity in the vertical direction of pixels having low brightness isdetermined. It should be noted that there is no need to always countonly pixels having brightness equal to a pixel corresponding to thelocal minimum value, since counting of the number of pixels is intendedto determine the continuity of pixels having low brightness. Inparticular, when the brightness is shown by a gradation classified into256 grades and the gradation of a pixel indicative of a local minimumvalue is 20, it is preferable to count the continuity with gradationhaving a width such as 20±5.

When the counted number of continuous pixels is larger than thepredetermined number, which is preliminarily set, in the step S119(S119: YES), the image processing device 2 determines that direction ofan area of the face of the driver is within a range of 22-43%, ofcandidates of a detection object as a detection object (S111) andupdates the content recorded in the HD 23 or the RAM 24 as a candidateof a detection object. In the step S111, whether the ratio of the lengthof the range in the horizontal direction of a detection area including adetection object, i.e. an area of the face, to the length of the rangein the horizontal direction of a candidate of a detection object, i.e.an area including nares to right and left nostrils, is within apredetermined range, a range of 22-43% here, or not is determined forall candidates of a detection object, and a candidate of a detectionobject, the ratio of which is within the predetermined range, isdetected as a detection object. It should be noted that a candidate of adetection object is specified on the basis of an index (transverse edgescore which will be explained later) to be used in a detecting processof a range shown in the steps S109-S110 when there are a plurality ofcandidates of a detection object the ratio of which is within thepredetermined range in the step S111. Moreover, the numerical value of22-43% shown as a predetermined range is not fixed and is a numericalvalue which will be set properly depending on factors such as the raceof the driver.

The image processing device 2 then sets a test area, which includes thedetection object specified in the step S111 and has a width in thehorizontal direction equal to the detection object and a the detectionobject specified in the step S111 is false and detection of a detectionobject is impossible under control of the CPU 21 (S117), updates therecord content of the HD 23 or the RAM 24 on the basis of the determinedresult and terminates the process. That is, it is determined that theframe of an eyeglass is detected falsely when the continuity of pixelshaving low brightness is larger than or equal to the predeterminednumber.

When the counted number of continuous pixels is smaller than or equal tothe predetermined number, which is preliminarily set, in the step S119(S119: NO), the image processing device 2 determines that the detectionobject specified in the step S111 is true and detection of a detectionobject has been achieved under control of the CPU 21 (S120), updates therecord content of the HD 23 or the RAM 24 on the basis of the determinedresult and terminates the process.

It should be noted that it is also determined that detection of adetection object is impossible when all candidates of a detection objectis excluded in the process of narrowing down candidates of a detectionobject, such as the steps S109-S111.

The following description will explain the process, which has beenexplained using the flow chart in Drawing 2 and Drawing 3, furtherspecifically. Drawing 4 is an explanatory view conceptually showing anexample of a process from decision of a range to detection of acandidate of a detection object of a process of an image of an imageprocessing system according to Embodiment 1 of the present invention.Drawing 4(a) shows a state where the range of a process of an image isdecided, wherein the outer frame shown in full line is the entire imageshown by image data extracted in the step S101, and a taken image of theface of the driver and an area including nares in the face of thedriver, which is a detection object, to right and left nostrils areshown. The lines in the vertical direction (Y-axis direction) of theimage shown in long dashed short dashed line are the range of the areadetected in the step S102, i.e., the width of the facial contour of thedriver. In addition, an area surrounded by the width of the facialcontour shown in long dashed short dashed line and the upper and lowerframe in the entire image shown in full line is the range of imageprocessing to be set in the step S103.

Drawing 4(b) is a graph showing the distribution of an integrated valueof the brightness in the vertical direction, which is obtained byintegrating the brightness of pixels lined up in the horizontaldirection in the step S104 and derived in the step S105. Drawing 4(b)shows the distribution of an integrated value of the brightness in thevertical direction of the image shown in Drawing 4(a), wherein theordinate axis indicates a coordinate in the vertical directioncorresponding to Drawing 4(a) and the abscissa axis indicates anintegrated value of the brightness. As shown in Drawing 4(b), anintegrated value of the brightness in the vertical direction changes soas to have local minimum values indicated by the arrows at parts such aseyebrows, eyes, nares and a mouth, and it can be understood that it ispossible to detect candidates of a detection object on the basis of thelocal minimum values in the step S106.

The following description will explain a process according to the stepsS109-S111. Drawing 5 is an explanatory view schematically showing anexample of a range for which an end detecting process of an imageprocessing system according to Embodiment 1 of the present invention isperformed, and Drawing 6 is an explanatory view showing an example ofcoefficients to be used in an end detecting process of an imageprocessing system according to Embodiment 1 of the present invention.Drawing 5 shows pixels in the vicinity of a candidate of a detectionobject, wherein the numbers shown on the upper side of Drawing 5indicate locations in the horizontal direction of pixels, i.e. xcoordinates, and the symbols shown on the left side indicate locationsin the vertical direction of pixels, i.e. y coordinates. The row y ofpixels, which are lined up in the lateral direction of Drawing 5 andshown with diagonal lines extending from top right to bottom left andhave a value y of a y coordinate, indicates a candidate of a detectionobject, and used for detection of an end are the row y of pixels, andthe row y+1 and the row y−1 of pixels, which adjoin the row y of pixelsfrom upside and downside, are shown with diagonal lines extending fromtop left to bottom right and have values of y coordinates respectivelyof y+1 and y−1. In addition, multiplication of each pixel included inthe rows y, y+1 and y−1 of pixels by the coefficients shown in Drawing 6functions as a transverse edge filter for clarifying an end of an areaof continuous pixels, which are lined up in the horizontal direction andhave low brightness. The coefficients shown in Drawing 6 indicatecoefficients to be used for multiplication of the brightness of ninepixels as a matrix of 3×3, so that the brightness of one pixel at thecenter and the brightness of eight adjacent pixels are respectivelymultiplied with one corresponding coefficient and the absolute value ofthe sum of the result is computed as a transverse edge coefficient of apixel located at the center. The transverse edge coefficient is obtainedby adding numerical values obtained by multiplying the brightness ofadjacent pixels on the left side by “−1” and numerical values obtainedby multiplying the brightness of adjacent pixels on the right side by“1” as shown in Drawing 6. It should be noted that the area of 3×3 shownin wide line in Drawing 5 shows a state where a transverse edge filtercorresponds to a pixel indicated by a coordinate (2, y+1), and thetransverse edge coefficient of the pixel indicated by the coordinate (2,y+1) is computed by the following Expression 2.|P(3, y+2)+P(3, y+1)+P(3, y)−P(1, y+2)−P(1, y+1)−P(1, y)|  Expression 2

Here, x: coordinate in the horizontal direction (x coordinate)

-   -   y: coordinate in the vertical direction (y coordinate)    -   p(x, y): brightness of pixel of coordinate (x, y)

As described above, a transverse edge coefficient is computed for pixelsincluded in the row y of pixels lined up in the horizontal direction,which is a candidate of a detection object, and the rows y−1 and y+1 ofpixels adjoining thereof from upward and downward.

Drawing 7 is an explanatory view schematically showing an example of arange for which an end detecting process of an image processing systemaccording to Embodiment 1 of the present invention is performed. Thenumbers shown on the upper side of Drawing 7 indicate locations in thehorizontal direction of pixels, i.e. x coordinates, and the symbolsshown on the left side indicate locations in the vertical direction ofpixels, i.e. y coordinates. The row y of pixels, which are lined up inthe lateral direction of Drawing 7 and have a value y of a y coordinate,indicates a candidate of a detection object, and a transverse edgecoefficient is computed for the row y of pixels, and the row y+1 and therow y−1 of pixels, which adjoin the row y of pixels from upside anddownside and have values of y coordinates respectively of y+1 and y−1.In addition, regarding each of pixels excluding the pixels at the endsof the row y of pixels, which are shown with diagonal lines extendingfrom top right to bottom left, the transverse edge coefficient and apredetermined threshold preliminarily set are compared for nine pixelsincluding one pixel and eight pixels adjacent to said pixel. Inaddition, an index, which indicates the number of pixels having atransverse edge coefficient larger than the predetermined threshold, iscomputed as a transverse edge score of the one pixel. The area of 3×3surrounded by wide line in Drawing 7 indicates pixels necessary forcomputation of a transverse edge score of a pixel indicated by acoordinate (3, y). The number of pixels, which has a transverse edgecoefficient larger than the threshold, of the nine pixels in the areasurrounded by wide line in Drawing 7 becomes a transverse edge score ofthe pixel indicated by the coordinate (3, y).

In addition, it is determined that a pixel, which has a transverse edgescore that may indicates a value from 0 to 9 is larger than or equal to5, is within the range in the horizontal direction of a candidate of adetection object. That is, the image processing device 2 detects theleftmost pixel having a transverse edge score larger than or equal to 5as a left end in the horizontal direction of a candidate of a detectionobject and a rightmost pixel having a transverse edge score larger thanor equal to 5 as a right end in the horizontal direction of a candidateof a detection object in the step S109. In addition, detected in thestep S110 is the range in the horizontal direction of a candidate of adetection object on the basis of the detected right and left ends.

Drawing 8 is an explanatory view schematically showing an example of arange for which an end detecting process of an image processing systemaccording to Embodiment 1 of the present invention is performed. Drawing8 shows pixels of a candidate of a detection object and transverse edgescores, and the numbers shown on the upper side of Drawing 8 indicatelocations in the horizontal direction of the pixels, i.e., xcoordinates. Detected for the example shown in Drawing 8 is a range inthe horizontal direction of a detection object from a pixel having an xcoordinate of 5 to a pixel having an x coordinate of 636.

Drawing 9 is an explanatory view showing candidates of a detectionobject of an image processing system according to Embodiment 1 of thepresent invention. Drawing 9 shows an image of the face of the driverand candidates of a detection object, the range in the horizontaldirection of which is detected in the process in the steps S109-S110. InDrawing 9, the x-marks indicate the right and left ends detected in thestep S109 and the line segments connecting the right and left endsindicated by the x-marks indicate candidates of a detection object. Acandidate, the right and left ends of which are not detected, isexcluded from a detection object at this stage. In the example shown inDrawing 9, the locations of the eyebrows, eyes, nares, mouth and jaw arethe candidates of a detection object.

In addition, a detection object is specified from the candidates of adetection object in the step S111 by comparing the range in thehorizontal direction of the area of the face of the driver shown inDrawing 4 and the range in the horizontal direction of the candidates ofa detection object shown in Drawing 9. It should be noted that, when aplurality of candidates of a detection object are specified in the stepS111, the number of pixels having a transverse edge score larger than orequal to a predetermined value in the respective pixels lined up in thehorizontal direction forming each specified detection object is countedfor each specified detection object, and an index indicated by theobtained numerical value is set as a naris area score. In addition, adetection object having the maximum naris area score is set as a truedetection object and the other detection objects are excluded as a falsedetection object.

Drawing 10 is an explanatory view conceptually showing a naris areascore of an image processing system according to Embodiment 1 of thepresent invention. Drawing 10 shows pixels included in the range in thehorizontal direction of the detection object detected in Drawing 8 andnumerical values indicating transverse edge scores of said pixels, andthe pixels with circled numerical values indicate that the transverseedge score is larger than or equal to a predetermined value, 5 here. Inaddition, the number of pixels having a transverse edge score largerthan or equal to a predetermined value is counted and set as a narisarea score.

In addition, in an image processing system of the present invention,true or false of the specified detection object is further determined bya process after the step S112.

Various kinds of conditions and the like including numerical valuesshown in the above Embodiment 1 are absolutely an example, and can beset properly depending on the situation such as the system structure andthe purpose.

Embodiment 2

Drawing 11 is a block diagram showing a structure example of an imageprocessing system according to Embodiment 2 of the present invention.Denoted at 1 in Drawing 11 is the image pickup device, which isconnected with the image processing device 2 via a dedicated cable, forexample. The image pickup device 1 comprises the MPU 11, the ROM 12, theRAM 13, the image pickup unit 14, the A/D converter 15, the frame memory16 and the communication interface 17.

The image processing device 2, which comprises the CPU 21, the auxiliaryrecording unit 22, the HD 23, the RAM 24, the frame memory 25 and thecommunication interface 26, reads various kinds of information, with theauxiliary recording unit 22, from a record medium 42 recording variouskinds of information such as data and a computer program 32 according toEmbodiment 2 of the present invention, records the same in the HD 23,records the same in the RAM 24 and runs the same with the CPU 21, so asto run various kinds of procedures according to Embodiment 2 of thepresent invention.

It should be noted that the detailed explanation of each device, whichis the same as Embodiment 1, is obtained by referring to Embodiment 1and will be omitted.

The following description will explain a process of various kinds ofdevices to be used in an image processing system according to Embodiment2 of the present invention. Embodiment 2 of the present invention is fora detection object of nares in the face of the driver from an imageobtained by image pickup of the face of the driver with the image pickupdevice 1 such as an in-vehicle camera which is mounted on a vehicle, forexample. Drawing 12 is a flow chart showing an example of a process ofthe image processing device 2 to be used in an image processing systemaccording to Embodiment 2 of the present invention. The image processingdevice 2 extracts image data, which is obtained by image pickup of theimage pickup device 1 and accepted via the communication interface 26,from the frame memory 25 under control of the CPU 21 for executing thecomputer program 32 recorded in the RAM 24 (S201), detects the width ofthe face of the driver, i.e. the range in the horizontal direction ofthe contour which is the boundary of an area indicative of the face,from the extracted image data by, for example, the contour widthdetecting process shown in Embodiment 1 (S202) and further detects anarea including nares to right and left nostrils as a naris peripheralarea (S203). The range in the horizontal direction of the contourdetected in the step S202 and the naris peripheral area detected in thestep S203 are recorded in the HD 23 or the RAM 24. The method shown inEmbodiment 1 is used as the method for detecting a naris peripheral areain the step S203, for example.

The image processing device 2 then derives two points, which becomelocal minimum values, as local minimum points from a change in thebrightness of pixels in the horizontal direction of the naris peripheralarea detected in the step S203 under control of the CPU 21 (S204) andsets a search range for detecting a detection object on the basis of thederived two local minimum points (S205). The search range set in thestep S205 is recorded in the HD 23 or the RAM 24. When there are threeor more local minimum points in the step S205, two points having smallerbrightness are derived as local minimum points. A range to be set in thestep S205 is a range including respectively 15 pixels on either side inthe horizontal direction and respectively 5 pixels on either side in thevertical direction for each of the two derived points of local minimumvalues. It should be noted that it can be considered that a search rangeis set in the step S205 for each of right and left nares since the twolocal minimum points derived in the step S204 are considered pointsrelating to right and left nares.

The image processing device 2 then converts the brightness of all pixelsin the search range set in the step S205 under control of the CPU 21 bya black area computation filtering process of performing addition basedon the brightness of another adjoining pixel and subtraction based onthe brightness of a pixel located at a predetermined distance in thehorizontal direction and the brightness of a pixel located at apredetermined distance in the vertical direction (S206), detects apixel, a converted value of which 15 becomes the minimum, as a detectionobject (S207) and records the detected result in the HD 23 or the RAM24. The process in the steps S206-S207 is performed for pixels includedin the respective search ranges of right and left nares.

The following description will explain the process, which has beenexplained using the flow chart in Drawing 12, further specifically.Drawing 13 is an explanatory view conceptually showing an example ofsetting of a detection range of an image processing system according toEmbodiment 2 of the present invention. In Drawing 13, the outer frameshown in full line is the entire image shown by image data extracted inthe step S201 and the lines in the vertical direction (Y-axis direction)of the image shown in long dashed short dashed line is the range of thearea detected in the step S202, i.e., the width of the facial contour ofthe driver. In addition, the line segment in the horizontal directionindicated by wide full line is an area including nares to right and leftnostrils, which is detected in the step S203 as a naris peripheral area.

Drawing 14 is an explanatory view conceptually showing an example ofsetting of a search range of an image processing system according toEmbodiment 2 of the present invention. Drawing 14 shows a peripheralimage including nares and the rectangular ranges shown in full line inDrawing 14 are search ranges which are set in the step S205 for each ofright and left nares.

Drawing 15 is an explanatory view showing an example of coefficients tobe used in a black area computation filtering process of an imageprocessing system according to Embodiment 2 of the present invention. Bymultiplying pixels included in the search range by coefficients shown inDrawing 15, it is possible to clarify an area where the brightness islow at the center and high in the surrounding. In the black areacomputation filter shown in Drawing 15, “1” is set as a coefficient tobe used for multiplication of the brightness of one pixel, which becomesan object of conversion, and the brightness of eight adjacent pixels,“−1” is set as a coefficient to be used for multiplication of thebrightness of two pixels lined up at a predetermined distance from theone pixel on each of right and left sides and “−1” is set as acoefficient to be used for multiplication of the brightness of twopixels lined up at a predetermined distance from the one pixel on eachof upper and lower sides. It should be noted that the predetermineddistance is set to 1/18 of the area detected in the step S202.

Drawing 16 is an explanatory view conceptually showing an example ofdetection using a black area computation filtering process of an imageprocessing system according to Embodiment 2 of the present invention.Drawing 16 shows the location of a black area computation filter in ablack area computation filtering process, which is performed for adetection object detected as a naris on your left side in Drawing 16, inthe search range set as shown in Drawing 14. Regarding the black areacomputation filter set with conditions shown in Drawing 15, coefficientsfor addition are located at nares and coefficients for subtraction arelocated outside nares in the black area computation filtering processfor pixels in the vicinity of the center of nares as shown in Drawing16, it is possible to clarify the center of nares.

Various kinds of conditions and the like including numerical valuesshown in the above Embodiment 2 are absolutely an example, and can beset properly depending on the situation such as the system structure andthe purpose.

Embodiment 3

Drawing 17 is a block diagram showing a structure example of an imageprocessing system according to Embodiment 3 of the present invention.Denoted at 1 in Drawing 17 is the image pickup device, which isconnected with the image processing device 2 via a dedicated cable, forexample. The image pickup device 1 comprises the MPU 11, the ROM 12, theRAM 13, the image pickup unit 14, the A/D converter 15, the frame memory16 and the communication interface 17.

The image processing device 2, which comprises the CPU 21, the auxiliaryrecording unit 22, the HD 23, the RAM 24, the frame memory 25 and thecommunication interface 26, reads various kinds of information, with theauxiliary recording unit 22, from a record medium 43 recording variouskinds of information such as data and a computer program 33 according toEmbodiment 3 of the present invention, records the same in the HD 23,records the same in the RAM 24 and runs the same with the CPU 21, so asto run various kinds of procedures according to Embodiment 3 of thepresent invention.

It should be noted that the detailed explanation of each device, whichis the same as Embodiment 1, is obtained by referring to Embodiment 1and will be omitted.

The following description will explain a process of various kinds ofdevices to be used in an image processing system according to Embodiment3 of the present invention. Embodiment 3 of the present invention is fora detection object of a downside area surrounding nares in the face ofthe driver from an image obtained by image pickup of the face of thedriver with the image pickup device 1 such as an in-vehicle camera whichis mounted on a vehicle, for example. Drawing 18 is a flow chart showingan example of a process of the image processing device 2 to be used inan image processing system according to Embodiment 3 of the presentinvention. The image processing device 2 extracts image data, which isobtained by image pickup of the image pickup device 1 and accepted viathe communication interface 26, from the frame memory 25 under controlof the CPU 21 for executing the computer program 33 recorded in the RAM24 (S301), detects the width of the face of the driver, i.e. the rangein the horizontal direction of the contour which is the boundary of anarea indicative of the face, from the extracted image data by, forexample, the contour width detecting process shown in Embodiment 1(S302), further detects the location of the eyes and nasal apex by aneyes and nasal apex detecting process using a process such as patternmatching (S303) and sets a search range on the basis of the detectedrange in the horizontal direction of the contour and the location of theeyes and nasal apex (S304). The range in the horizontal direction of thecontour detected in the step S302, the location of the eyes and nasalapex detected in the step S303 and the search range set in the step S304are recorded in the HD 23 or the RAM 24. It should be noted that thedetailed content of the eyes and nasal apex detecting process in thestep S302 is described in documents such as Japanese Patent ApplicationLaid-Open Nos. 2004-234367 and 2004-234494 which have been applied bythe present applicant, for example. Set as the search range in the stepS304 is an area based on, for example, an upper end in the verticaldirection located lower than the average of y coordinates indicative oflocations in the vertical direction of the eyes by a distance equal to1/16 of the width in the horizontal direction of the contour, a lowerend located lower than the average of y coordinates of the eyes by adistance equal to ⅜ of the width of the contour, a left end in thehorizontal direction located to the left of an x coordinate indicativeof the location in the horizontal direction of the nasal apex by adistance equal to ⅛ of the width of the contour and a right end locatedto the right of an x coordinate of the nasal apex by a distance equal to⅛ of the width of the contour.

The image processing device 2 then derives a horizontal pixel score,which is an index obtained by multiplying a numerical value based on abrightness difference from pixels adjacent in the horizontal directionby a numerical value indicative of a low level of brightness of pixelsadjacent in the horizontal direction, for all pixels in the search rangeset in the step S304, under control of the CPU 21 (S305) and integratesderived horizontal pixel scores in the vertical direction so as toderive a horizontal score which is an index indicative of a change in anintegrated value in the horizontal direction (S306). The imageprocessing device 2 further derives a vertical pixel score, which is anindex obtained by multiplying a numerical value based on a brightnessdifference from pixels adjacent in the vertical direction by a numericalvalue indicative of a low level of brightness of pixels adjacent in thevertical direction, for all pixels in the search range, under control ofthe CPU 21 (S307) and integrates derived vertical pixel scores in thehorizontal direction so as to derive a vertical score which is an indexindicative of a change in an integrated value in the vertical direction(S308). The image processing device 2 then detects an area based on arange in the horizontal direction from the maximum value to the minimumvalue of the horizontal score derived in the step S306 and a range inthe vertical direction from the maximum value to the minimum value ofthe vertical score derived in the step S307 as a detection object undercontrol of the CPU 21 (S309) and records the detected result in the HD23 or the RAM 24.

The following description will explain the process, which has beenexplained using the flow chart in Drawing 18, further specifically.Drawing 19 is an explanatory view conceptually showing an example ofsetting of a search range of an image processing system according toEmbodiment 3 of the present invention. In Drawing 19, the outer frameshown in full line is the entire image shown by image data extracted inthe step S301, the lines in the vertical direction (Y-axis direction) ofthe image shown in long dashed short dashed line is the range of thearea detected in the step S302, i.e. the width of the facial contour ofthe driver, and the positions indicated by the x-marks are the locationsof the eyes and nasal apex detected in the step S303. In addition, thearea shown in dotted line is the search range to be set in the stepS304.

The horizontal pixel score to be derived in the step S305 is shown bythe following Expression 3. $\begin{matrix}{{{Sh}\left( {x,y} \right)} = \left\{ \begin{matrix}{{{H\left( {x,y} \right)} \cdot \left( {255 - {P\left( {{x + 1},y} \right)}} \right)};\left( {{H\left( {x,y} \right)} \geqq 0} \right)} \\{{{H\left( {x,y} \right)} \cdot \left( {255 - {P\left( {{x - 1},y} \right)}} \right)};\left( {{H\left( {x,y} \right)} < 0} \right)}\end{matrix} \right.} & {{Expression}\quad 3}\end{matrix}$

Here, x: coordinate in the horizontal direction (x coordinate)

-   -   y: coordinate in the vertical direction (y coordinate)    -   Sh(x, y): horizontal pixel score of pixel of coordinate (x, y)    -   H(x, y): horizontal edge filtering process result of pixel of        coordinate (x, y)    -   P(x, y): brightness of pixel of coordinate (x, y)

Drawing 20 is an explanatory view showing an example of coefficients tobe used in a horizontal edge filtering process of an image processingsystem according to Embodiment 3 of the present invention. In themathematical expression shown in Expression 3, H(x, y) indicates theresult of a horizontal edge filtering process to be performed using thecoefficients shown in Drawing 20. Drawing shows coefficients to be usedfor multiplication of the brightness of nine pixels as a matrix of 3×3,so that the brightness of one pixel at the center and the brightness ofeight adjacent pixels are respectively multiplied by a correspondingcoefficient and the sum of the result is computed as a horizontal edgefiltering process result of a pixel located at the center. In thehorizontal edge filtering process to be performed using the coefficientsshown in Drawing 20, an index which is the result is obtained by addingnumerical values obtained by multiplying the brightness of adjacentpixels on the left side by “1” and numerical values obtained bymultiplying the brightness of adjacent pixels on the right side by “−1”.That is, a numerical value based on a brightness difference from pixelsadjacent in the horizontal direction is obtained by the horizontal edgefiltering process.

A mathematical expression to be used for multiplication of the result ofthe horizontal edge filtering process is a value obtained by subtractingthe brightness of pixels adjacent in the horizontal direction from “255”indicative of the maximum numerical value of the gray value of thebrightness of a pixel classified into 256 grades, and is a numericalvalue indicative of a low level of brightness. As described above, thehorizontal pixel score is an index based on a numerical value based on abrightness difference from pixels adjacent in the horizontal directionand a numerical value indicative of a low level of brightness of pixelsadjacent in the horizontal direction. It should be noted that thepixels, the brightness of which is used for a mathematical expression tobe used for multiplication of the result of the horizontal edgefiltering process, depends on the positive and negative of the result ofthe horizontal edge filtering process.

In addition, the horizontal score to be derived in the step S306 is anindex indicative of the relation between the x coordinate, which is thelocation in the horizontal direction to be derived by integratinghorizontal pixel scores in the vertical direction, and an integratedvalue. That is, a change in the horizontal direction in the horizontalpixel score is indicated by the horizontal score. In particular, thehorizontal score can be shown so that the value becomes large at aposition where the brightness of a pixel in the vertical directiongreatly decreases and the value becomes small at a position where thebrightness of a pixel greatly increases.

The vertical pixel score to be derived in the step S307 is shown by thefollowing Expression 4. $\begin{matrix}{{{Sv}\left( {x,y} \right)} = \left\{ \begin{matrix}{{{V\left( {x,y} \right)} \cdot \left( {255 - {P\left( {x,{y + 1}} \right)}} \right)};\left( {{V\left( {x,y} \right)} \geqq 0} \right)} \\{{{V\left( {x,y} \right)} \cdot \left( {255 - {P\left( {x,{y - 1}} \right)}} \right)};\left( {{V\left( {x,y} \right)} < 0} \right)}\end{matrix} \right.} & {{Expression}\quad 4}\end{matrix}$

Here, Vh(x, y): vertical pixel score of pixel of coordinate (x, y)

-   -   V(x, y): vertical edge filtering process result of pixel of        coordinate (x, y)    -   P(x, y): brightness of pixel of coordinate (x, y)

Drawing 21 is an explanatory view showing an example of coefficients tobe used in a vertical edge filtering process of an image processingsystem according to Embodiment 3 of the present invention. In themathematical expression shown in Expression 4, V(x, y) indicates theresult of a vertical edge filtering process to be performed using thecoefficients shown in Drawing 21. Drawing 21 shows coefficients to beused for multiplication of the brightness of nine pixels as a matrix of3×3, so that the brightness of one pixel at the center and thebrightness of eight adjacent pixels are respectively multiplied by acorresponding coefficient and the sum of the result is computed as avertical edge filtering process result of a pixel located at the center.In the vertical edge filtering process to be performed using thecoefficients shown in Drawing 21, an index which is the result isobtained by adding numerical values obtained by multiplying thebrightness of adjacent pixels on the upper side by “1” and numericalvalues obtained by multiplying the brightness of adjacent pixels on thelower side by “−1”. That is, a numerical value based on a brightnessdifference from pixels adjacent in the vertical direction is obtained bythe vertical edge filtering process.

A mathematical expression to be used for multiplication of the result ofthe vertical edge filtering process is a value obtained by subtractingthe brightness of pixels adjacent in the vertical direction from “255”indicative of the maximum numerical value of the gray value of thebrightness of a pixel classified into 256 grades, and is a numericalvalue indicative of a low level of brightness. As described above, thevertical pixel score is an index based on a numerical value based on abrightness difference from pixels adjacent in the vertical direction anda numerical value indicative of a low level of brightness of pixelsadjacent in the vertical direction. It should be noted that the pixels,the brightness of which is used for a mathematical expression to be usedfor multiplication of the result of the vertical edge filtering process,depends on the positive and negative of the result of the vertical edgefiltering process.

In addition, the vertical score to be derived in the step S308 is anindex indicative of the relation between the y coordinate, which is thelocation in the vertical direction to be derived by integrating verticalpixel scores in the horizontal direction, and an integrated value. Thatis, a change in the vertical direction in the vertical pixel score isindicated by the vertical score. In particular, the vertical score canbe shown so that the value becomes large at a position where thebrightness of a pixel in the vertical direction greatly decreases andthe value becomes small at a position where the brightness of a pixelgreatly increases.

In addition, detected in the step S309 is an area having the maximumvalue of the horizontal score at the left end in the horizontaldirection, the minimum value at the right end in the horizontaldirection, the maximum value of the vertical score at the upper end inthe vertical direction and the minimum value at the lower end in thevertical direction.

Drawing 22 is an explanatory view showing a result of detection of animage processing system according to Embodiment 3 of the presentinvention. Drawing 22 shows the search range set in the step S304, andthe rectangular area shown with diagonal lines surrounded by the leftend L, the right end R, the upper end U and the lower end D is thedetected downside area surrounding nares.

Various kinds of conditions and the like including numerical valuesshown in the above Embodiment 3 are absolutely an example, and can beset properly depending on the situation such as the system structure andthe purpose. For example, though the above Embodiment 3 shows anembodiment of deriving the vertical score after deriving the horizontalscore, a variety of embodiments may be developed such as deriving thehorizontal score after deriving the vertical score, or deriving only thevertical score and setting the left end and the right end in thehorizontal direction as the width of the facial contour without derivingthe horizontal score.

Embodiment 4

Drawing 23 is a block diagram showing a structure example of an imageprocessing system according to Embodiment 4 of the present invention.Denoted at 1 in Drawing 23 is the image pickup device, which isconnected with the image processing device 2 via a dedicated cable, forexample. The image pickup device 1 comprises the MPU 11, the ROM 12, theRAM 13, the image pickup unit 14, the A/D converter 15, the frame memory16 and the communication interface 17.

The image processing device 2, which comprises the CPU 21, the auxiliaryrecording unit 22, the HD 23, the RAM 24, the frame memory 25 and thecommunication interface 26, reads various kinds of information, with theauxiliary recording unit 22, from a record medium 44 recording variouskinds of information such as data and a computer program 34 according toEmbodiment 4 of the present invention, records the same in the HD 23,records the same in the RAM 24 and runs the same with the CPU 21, so asto run various kinds of procedures according to Embodiment 4 of thepresent invention. Recorded in the image processing device 2 accordingto Embodiment 4 of the present invention is a plurality of detectionmethods including the detection methods explained in Embodiments 1 to 3of the present invention.

It should be noted that the detailed explanation of each device, whichis the same as Embodiment 1, is obtained by referring to Embodiment 1and will be omitted.

The following description will explain a process of various kinds ofdevices to be used in an image processing system according to Embodiment4 of the present invention. Embodiment 4 of the present invention is fora detection object of an area including nares of the face of the driverfrom an image obtained by image pickup of the face of the driver withthe image pickup device 1 such as an in-vehicle camera which is mountedon a vehicle, for example. Drawing 24 is a flow chart showing an exampleof a process of the image processing device 2 to be used in an imageprocessing system according to Embodiment 4 of the present invention.The image processing device 2 extracts image data, which is obtained byimage pickup of the image pickup device 1 and accepted via thecommunication interface 26, from the frame memory 25 under control ofthe CPU 21 for executing the computer program 34 recorded in the RAM 24(S401), computes a mean value of the brightness of pixels included inthe extracted image data (S402) and compares the mean value, which isthe computed result, with a threshold which is preliminarily set for amean value of the brightness (S403). The image processing device 2further computes a variance value of the brightness of pixels undercontrol of the CPU 21 (S404) and compares the variance value, which isthe computed result, with a threshold which is preliminarily set for avariance value of the brightness (S405). The image processing device 2then decides the priority of a plurality of detection methods recordedin the HD 23 on the basis of the comparison result between the meanvalue of the brightness and the threshold and the comparison resultbetween the variance value of the brightness and the threshold undercontrol of the CPU 21 (S406).

In Embodiment 4 of the present invention, a process of deciding thepriority of detection methods on the basis of the mean value and thevariance value of the brightness is performed. The priority to bedecided is the necessity of execution of a plurality of detectionmethods and the order of execution, and the detection methods explainedabove in Embodiments 1 to 3 of the present invention and yet anotherdetection method are executed according to the decided content.

Since the mean value and the variance value of the brightness of imagedata greatly changes depending on the status of a light ray forirradiating the face of the driver, the status of irradiation isdetermined from the mean value and the variance value so as to selectthe most suitable detection method. In particular, when there arises apartial change such as irradiation of only the left half of the facewith sunlight, it is possible to determine that there arises a partialchange since the mean value of the 10 brightness becomes smaller than orequal to the threshold and the variance value of the brightness becomeslarger than or equal to the threshold, and priority is given to adetection method which is the most resistant to influence of the partialchange.

For example, in a state where there is no partial change, the narisperipheral area is detected by the detection method shown in Embodiment1 and nares are detected by the detection method shown in Embodiment 2using the detected result. Since the range to be the object of imageprocessing is limited by detecting the naris peripheral area, theprocessing speed is enhanced and the detection accuracy is enhanced.However, when a partial change arises, the reliability of the processusing a transverse edge filter in Embodiment 1 is reduced since there isa high possibility that brightness saturation arises. Accordingly,priority is given to detection of nares by the detection method shown inEmbodiment 2.

Though one threshold of the mean value of the brightness and onethreshold of the variance value of the brightness are set in the aboveEmbodiment 4, the present invention is not limited to this, and aplurality of thresholds may be set and the priority of detection methodsdepending on a variety of situations may be decided, or processingconditions such as various kinds of set values necessary for imageprocessing for detection may be decided on the basis of the mean valueand the variance value.

Though a process for image data indicated by a plane rectangularcoordinate system is shown in the above Embodiments 1 to 4, the presentinvention is not limited to this and can be, for example, applied toimage data of a variety of coordinate systems such as application toimage data indicated by a coordinate system in which the first directionand the second direction cross at an angle of 60 degrees, when dealingwith an image including pixels arranged in honeycomb geometry.

Moreover, though an embodiment for a detection object of the driver of avehicle is shown in the above Embodiments 1 to 4, the present inventionis not limited to this and an embodiment for a detection object of avariety of human, or even a living matter other than human or anonliving matter can be also employed.

Furthermore, though an embodiment for detecting a detection object froman image generated by image pickup with an image pickup device using anin-vehicle camera is shown in the above Embodiments 1 to 4, the presentinvention is not limited to this and can be applied to a variety ofimage processing for recording an image generated by a variety ofmethods with a variety of devices in an HD and detecting a specificdetection object from the recorded image.

1-21. (canceled)
 22. An image processing method of detecting a specificdetection object from a two-dimensional image in which pixels are linedup in both a first direction and a second direction that are differentfrom each other comprising: integrating brightness of pixels lined up inthe first direction; deriving a change in an integrated value in thesecond direction; detecting a plurality of locations in the seconddirection as locations corresponding to candidates of a detection objecton the basis of the derived change in an integrated value; detecting arange in the first direction on the basis of brightness of a pixel as acandidate of the detection object for each of rows of pixels lined up inthe first direction corresponding to each detected location; andspecifying the detection object from candidates of the detection objecton the basis of length of the detected range.
 23. An image processingmethod of detecting a specific detection object from a two-dimensionalimage in which pixels are lined up in both a first direction and asecond direction that are different from each other, comprising:converting brightness of one pixel on the basis of a result of bothaddition based on brightness of each other adjoining pixel andsubtraction based on both brightness of a pixel located at apredetermined distance in the first direction and brightness of a pixellocated at the predetermined distance in the second direction from saidone pixel; and detecting a detection object on the basis of a result ofconversion.
 24. An image processing method of detecting a specificdetection object from a two-dimensional image in which pixels are linedup in both a first direction and a second direction that are differentfrom each other, comprising: integrating numerical values in the seconddirection, said numerical values being based on a change in brightnessof pixels lined up in the first direction; deriving a change in anintegrated value in the first direction; integrating numerical values inthe first direction, said numerical values being based on a change inbrightness of pixels lined up in the second direction; deriving a changein an integrated value in the second direction; and detecting adetection object on the basis of both a range in the first direction onthe basis of the derived change in an integrated value in the firstdirection and a range in the second direction on the basis of thederived change in an integrated value in the second direction.
 25. Animage processing method of detecting a specific detection object from atwo-dimensional image including a plurality of pixels by a plurality ofdetection methods, comprising: computing a mean value of brightness ofpixels; computing a variance value of brightness of pixels; and decidingpriority of each of the plurality of detection methods on the basis ofboth the computed mean value and the computed variance value.
 26. Animage processing device which detects a specific detection object from atwo-dimensional image in which pixels are lined up in both a firstdirection and a second direction that are different from each other,comprising: a deriving part which integrates brightness of pixels linedup in the first direction and derives a change in an integrated value inthe second direction; a candidate detecting part which detects aplurality of locations in the second direction as locationscorresponding to candidates of a detection object on the basis of thederived change in the integrated value; a range detecting part whichdetects a range in the first direction on the basis of brightness of apixel as a candidate of the detection object for each of rows of pixelslined up in the first direction corresponding to each of the pluralityof detected locations; and a specifying part which specifies thedetection object from candidates of the specific detection object on thebasis of length of the detected range.
 27. The image processing deviceaccording to claim 26, wherein the candidate detecting part detects theplurality of locations for which a change in an integrated value ofbrightness of pixels integrated in the first direction indicates a localminimum value.
 28. The image processing device according to claim 27,further comprising a part which obtains a quadratic differential valueof the change in an integrated value derived by the deriving part,wherein the candidate detecting part detects a predetermined number oflocations in the plurality of locations indicating a local minimumvalue, on the basis of a result of quadratic differential
 29. The imageprocessing device according to claim 26, wherein the range detectingpart detects a range on the basis of a change in brightness of pixelslined up in the first direction.
 30. The image processing deviceaccording to claim 26, further comprising a part which detects a rangein the first direction of a detection area which includes the detectionobject and has a range in the first direction larger than the detectionobject, wherein the specifying part specifies the detection object onthe basis of a result of comparison between length of a range in thefirst direction of a candidate of the detection object detected by therange detecting part and length of a range in the first direction of adetection area including the detection object.
 31. The image processingdevice according to claim 26, further comprising: a part whichintegrates brightness of pixels lined up in the second directionaccording to the specified detection object and derives a change in anintegrated value in the first direction; a local minimum value detectingpart which detects a local minimum value from a change in an integratedvalue in the first direction; a part which counts the number of detectedlocal minimum values; and a part which determines that the specifieddetection object is false when the counted number is smaller than apredetermined number.
 32. The image processing device according to claim37, further comprising a part which determines that the specifieddetection object is false when the number of continuous pixels in thesecond direction is larger than a predetermined number, said continuouspixels both including a pixel corresponding to the local minimum valuedetected by the local minimum value detecting part and having brightnessequal to said pixel.
 33. An image processing device which detects aspecific detection object from a two-dimensional image in which pixelsare lined up in both a first direction and a second direction that aredifferent from each other, comprising: a part which converts brightnessof one pixel on the basis of a result of both addition based onbrightness of each other adjoining pixel and subtraction based on bothbrightness of a pixel located at a predetermined distance in the firstdirection and brightness of a pixel located at a predetermined distancein the second direction from said one pixel; and a detecting part whichdetects a detection object on the basis of a result of conversion. 34.The image processing device according to claim 33, wherein the detectingpart detects a pixel having a minimum converted value as the detectionobject.
 35. An image processing device which detects a specificdetection object from a two-dimensional image in which pixels are linedup in both a first direction and a second direction that are differentfrom each other, comprising: a first deriving part which integratesnumerical values, in the second direction, based on a change inbrightness of pixels lined up in the first direction and derives achange in an integrated value in the first direction; a second derivingpart which integrates numerical values, in the first direction, based ona change in brightness of pixels lined up in the second direction andderives a change in an integrated value in the second direction; and adetecting part which detects a detection object on the basis of both arange in the first direction based on the change in an integrated valuein the first direction derived by the first deriving part and a range inthe second direction based on the change in an integrated value in thesecond direction derived by the second deriving part.
 36. The imageprocessing device according to claim 35, wherein the first deriving partintegrates indexes, in the second direction, based on both a numericalvalue based on a brightness difference from an adjoining pixel in thefirst direction and a numerical value indicating a low level ofbrightness of the adjoining pixel in the first direction and derive achange in an integrated value in the first direction, wherein the secondderiving part integrates indexes, in the first direction, based on botha numerical value based on a brightness difference from an adjoiningpixel in the second direction and a numerical value indicating a lowlevel of brightness of the adjoining pixel in the second direction andderive a change in an integrated value in the second direction, andwherein the detecting part detects a detection object on the basis ofboth a range in the first direction from a location where the integratedvalue derived by the first deriving part becomes maximum to a locationwhere the integrated value derived by the first deriving part becomesminimum and a range in the second direction from a location where theintegrated value derived by the second deriving part becomes maximum toa location where the integrated value derived by the second derivingpart becomes minimum.
 37. An image processing device which detects aspecific detection object from a two-dimensional image including aplurality of pixels by a plurality of detection methods, comprising: apart which computes a mean value of brightness of a pixel; a part whichcomputes a variance value of brightness of a pixel; and a part whichdecides priority of each of the plurality of detection methods on thebasis of both the computed mean value and the computed variance value.38. An image processing system, comprising: an image processing deviceaccording to claim 26; and an image pickup device which generates animage to be processed by the image processing device, wherein thedetection object is an area including nares of a person in an imagetaken by the image pickup device, the first direction is a horizontaldirection, and the second direction is a vertical direction.
 39. Animage processing system, comprising: an image processing deviceaccording to claim 33; and an image pickup device which generates animage to be processed by the image processing device, wherein thedetection object is an area including nares of a person in an imagetaken by the image pickup device, the first direction is a horizontaldirection, and the second direction is a vertical direction.
 40. Animage processing system, comprising: an image processing deviceaccording to claim 35; and an image pickup device which generates animage to be processed by the image processing device, wherein thedetection object is an area including nares of a person in an imagetaken by the image pickup device, the first direction is a horizontaldirection, and the second direction is a vertical direction.
 41. Animage processing system, comprising: an image processing deviceaccording to claim 37; and an image pickup device which generates animage to be processed by the image processing device, wherein thedetection object is an area including nares of a person in an imagetaken by the image pickup device, the first direction is a horizontaldirection, and the second direction is a vertical direction.
 42. Acomputer-readable storage encoding a computer program that when executedcauses a computer to perform a method of detecting a specific detectionobject from a two-dimensional image in which pixels are lined up in botha first direction and a second direction that are different from eachother, said method comprising: integrating brightness of pixels lined upin the first direction; deriving a change in an integrated value in thesecond direction; detecting a plurality of locations in the seconddirection as locations corresponding to candidates of a detection objecton the basis of the derived change in an integrated value; detecting arange in the first direction on the basis of brightness of a pixel as acandidate of the detection object for each of rows of pixels lined up inthe first direction corresponding to each detected location; andspecifying the detection object from candidates of the detection objecton the basis of length of the detected range.
 43. A computer-readablestorage encoding a computer program that when executed causes a computerto perform a method of detecting a specific detection object from atwo-dimensional image in which pixels are lined up in both a firstdirection and a second direction that are different from each other,said method comprising: converting brightness of one pixel on the basisof a result of both addition based on brightness of each other adjoiningpixel and subtraction based on both brightness of a pixel located at apredetermined distance in the first direction and brightness of a pixellocated at a predetermined distance in the second direction from saidone pixel; and detecting the detection object on the basis of a resultof conversion.
 44. A computer-readable storage encoding a computerprogram that when executed causes a computer to perform a method ofdetecting a specific detection object from a two-dimensional image inwhich pixels are lined up in both a first direction and a seconddirection that are different from each other, said method comprising:integrating numerical values, in the second direction, based on a changein brightness of pixels lined up in the first direction; deriving achange in an integrated value in the first direction; integratingnumerical values, in the first direction, based on a change inbrightness of pixels lined up in the second direction; deriving a changein an integrated value in the second direction; and detecting adetection object on the basis of both a range in the first directionbased on the derived change in an integrated value in the firstdirection and a range in the second direction based on the derivedchange in an integrated value in the second direction.
 45. Acomputer-readable memory storage encoding a computer program that whenexecuted causes a computer to perform a method of detecting a specificdetection object from a two-dimensional image including a plurality ofpixels by a plurality of detection methods, said method comprising:computing a mean value of brightness of a pixel; computing a variancevalue of brightness of a pixel; and deciding priority of detectionmethods on the basis of both the computed mean value and the computedvariance value.